Most personal computers (PCs) employ a monitor or display screen to display text and graphics to a user. Touch screens or other touch-sensitive displays, such as those used on Tablet PCs, enable a more interactive and richer experience for a user by responding to an object such as a stylus that is brought near or into contact with the surface of the display screen, to enable the user to make selections of items that are displayed and to move such items around on the display. Although tablets greatly improve the functionality of a simple display screen, they are still generally limited in their response to an object.
The prior art also includes other types of interactive display systems that enable multiple objects placed on a display surface to be recognized and to respond to such objects. For example, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14–17. The metaDESK includes a near-horizontal graphical surface used to display two-dimensional (2D) geographical information. Above the graphical surface is disposed an arm-mounted flat-panel display that serves as an “active lens” for use in displaying three-dimensional (3D) geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface. The IR camera in the metaDESK can detect a reflection from an IR reflective material applied to the undersurface of passive objects called “phicons” that are placed on the graphical surface. In response to the IR camera detecting the IR reflective material (which is transparent to visible light) applied to the bottom of a “Great Dome phicon,” a map of the MIT campus is displayed on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located. Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user.
Also included with the metaDESK is a “passive lens” that is constructed of a 1 cm thick fiber-optic cluster material formed as a circle of about 12 cm in diameter and supported in a ring formed by a wooden frame that is provided with a handle. The image displayed on the metaDESK can include a circular region that is projected onto the display surface at the location of the passive lens, so that the portion of the displayed image under the passive lens is conveyed through the fiber-optic cluster material and appears on the upper surface of the passive lens. The article reports that earlier passive lenses that were made without any lens material or by simply using clear PLEXIGLAS™ were not successful in sustaining the illusion that the passive lens is a separate screen. Use of the fiber-optic cluster material was apparently more successful in providing this illusion.
When the passive lens is moved over the display surface of the metaDESK, the portion of the displayed image that is to appear on the upper surface of the passive lens is moved to track with the current location of the passive lens. In an initial embodiment, the location of the passive lens on the display was apparently detected and tracked using the IR camera vision sensing system. However, the article relates that subsequently, “a ‘Flock of Birds’ sensor was used to provide faster, more precise graphics updates.” (The FLOCK OF BIRDS™ sensor is a pulsed direct current (DC) magnetic tracking sensor sold by Ascension Technology Corporation.) The magnetic-field position sensor (FLOCK OF BIRDS™) and electrical-contact sensors are included in the metaDESK.
Although the prior art passive lens developed for use with the metaDESK appears to shift an image from the display surface to the upper surface of the passive filter, there is no provision for making the image projected on the display surface appear to be within the interior of an object. Similarly, there is no provision for making the image appear on another surface of an object, such as the side of a cube. When using the object while playing a game, it would be useful to display information or images that are only visible to a player on one side of the display surface, but not visible to players on the other sides of the display surface. Accordingly, it would be desirable to have the image appear on a side of the cube and thus, not readily visible to the other players located around the display surface. In addition, the prior art does not indicate how the passive lens of the metaDESK can be made to be interactive with the user or to respond to interactive gestures by the user. For example, it would be desirable to enable a user to select and move a graphic icon or other image toward an object positioned on the display surface and when the icon reaches the object, convey the graphic icon or image to a surface that is on or within the object. The graphic icon or other image would thus appear to be within or on a surface of the object. Again, the prior art does not discuss this form of interaction.
Another interactive display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc. in collaboration with others. These papers briefly describe a “HoloWall” and a “HoloTable,” both of which use IR light to detect objects that are proximate to or in contact with a display surface on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects become more clearly visible as they approach and then contact the panel. The objects thus detected can be a user's fingers or hands, or other objects. One of these papers, entitled “DataTiles: A Modular Platform for Mixed Physical and Graphical Interactions,” by J. Rekimoto et al., SIGCHI'01, Mar. 31, 2001, explains how objects tagged with a radio frequency identifier (RFID) and referred to as “tiles” can serve various functions when placed on a surface of a flat panel display that is able to read the RFID of the tile and respond in several different ways. For example, graphic data can be displayed and seen “within” the tiles (actually, an image on the display surface is viewed through the transparent tiles). However, there is no teaching or suggestion of how to convey an image displayed under an object, so that the image appears on or truly within the object. Accordingly, it will be apparent that an object and display system providing a richer experience and capable of conveying an image to a different surface of an object would be preferable, compared to the prior art approach.